Installation¶
Just pip install:
pip install omegaconf
If you want to try this notebook after checking out the repository be sure to run
python setup.py develop at the repository root before running this code.
xxxxxxxxxxfrom omegaconf import OmegaConfconf = OmegaConf.create()print(conf)From a dictionary¶
xxxxxxxxxxconf = OmegaConf.create(dict(k='v',list=[1,dict(a='1',b='2')]))print(OmegaConf.to_yaml(conf))From a list¶
xxxxxxxxxxconf = OmegaConf.create([1, dict(a=10, b=dict(a=10))])print(OmegaConf.to_yaml(conf))From a yaml file¶
xxxxxxxxxxconf = OmegaConf.load('../source/example.yaml')print(OmegaConf.to_yaml(conf))From a yaml string¶
xxxxxxxxxxyaml = """a: bb: clist:- item1- item2"""conf = OmegaConf.create(yaml)print(OmegaConf.to_yaml(conf))From a dot-list¶
xxxxxxxxxxdot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]conf = OmegaConf.from_dotlist(dot_list)print(OmegaConf.to_yaml(conf))From command line arguments¶
To parse the content of sys.arg:
xxxxxxxxxx# Simulating command line argumentsimport syssys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']conf = OmegaConf.from_cli()print(OmegaConf.to_yaml(conf))Access and manipulation¶
Input yaml file:
xxxxxxxxxxconf = OmegaConf.load('../source/example.yaml')print(OmegaConf.to_yaml(conf))Object style access:¶
xxxxxxxxxxconf.server.portdictionary style access¶
xxxxxxxxxxconf['log']['rotation']items in list¶
xxxxxxxxxxconf.users[0]Changing existing keys¶
xxxxxxxxxxconf.server.port = 81Adding new keys¶
xxxxxxxxxxconf.server.hostname = "localhost"Adding a new dictionary¶
xxxxxxxxxxconf.database = {'hostname': 'database01', 'port': 3306}providing default values¶
xxxxxxxxxxconf.missing_key or 'a default value'conf.get('missing_key', 'a default value')Accessing mandatory values¶
Accessing fields with the value ??? will cause a MissingMandatoryValue exception. Use this to indicate that the value must be set before accessing.
xxxxxxxxxximport pytestfrom omegaconf import MissingMandatoryValuewith pytest.raises(MissingMandatoryValue): conf.log.filexxxxxxxxxx# Variable interpolationOmegaConf support variable interpolation, Interpolations are evaluated lazily on access.### Config node interpolationxxxxxxxxxx# Variable interpolationOmegaConf support variable interpolation, Interpolations are evaluated lazily on access.## Config node interpolationInterpolations are evaluated lazily on field access.
Note below that when printed the interpolations are not resolved.
They get resolved once you access them.
xxxxxxxxxxconf = OmegaConf.load('../source/config_interpolation.yaml')print(OmegaConf.to_yaml(conf))xxxxxxxxxx# Primitive interpolation types are inherited from the referenced valueprint("conf.client.server_port: ", conf.client.server_port, type(conf.client.server_port).__name__)# Composite interpolation types are always stringprint("conf.client.url: ", conf.client.url, type(conf.client.url).__name__)xxxxxxxxxxto_yaml will resolve interpolation if `resolve=True` is passedxxxxxxxxxx`to_yaml()` will resolve interpolations if `resolve=True` is passedxxxxxxxxxxprint(OmegaConf.to_yaml(conf, resolve=True))xxxxxxxxxxInterpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:xxxxxxxxxxcfg = OmegaConf.create("""plans: A: plan A B: plan Bselected_plan: Aplan: ${plans.${selected_plan}}""")print(f"Default: cfg.plan = {cfg.plan}")cfg.selected_plan = "B"print(f"After selecting plan B: cfg.plan = {cfg.plan}")Default: cfg.plan = plan A After selecting plan B: cfg.plan = plan B
xxxxxxxxxx### Environment variable interpolationEnvironment variable interpolation is also supported.Input yaml file:xxxxxxxxxx## Environment variable interpolationEnvironment variable interpolation is also supported.xxxxxxxxxx# Let's set up the environment first (Only needed for this demonstration)import osos.environ['USER'] = 'omry'xxxxxxxxxx# Let's set up the environment first (only needed for this demonstration)import osos.environ['USER'] = 'omry'Here is an example config file interpolates with the USER environment variable:
xxxxxxxxxxconf = OmegaConf.load('../source/env_interpolation.yaml')print(OmegaConf.to_yaml(conf))xxxxxxxxxxconf = OmegaConf.load('../source/env_interpolation.yaml')print(OmegaConf.to_yaml(conf, resolve=True))xxxxxxxxxxYou can specify a default value to use in case the environment variable is not defined. The following example sets `abc123` as the the default value when `DB_PASSWORD` is not defined.xxxxxxxxxxos.environ.pop('DB_PASSWORD', None) # ensure env variable does not existcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})print(repr(cfg.database.password))OmegaConf.clear_cache(cfg) # clear resolver cacheos.environ["DB_PASSWORD"] = 'secret'print(repr(cfg.database.password))'abc123' 'secret'
xxxxxxxxxxEnvironment variables are parsed when they are recognized as valid quantities that may be evaluated (e.g., int, float, dict, list):xxxxxxxxxxcfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}', 'user': 'someuser', 'port': '${env:DB_PORT,3306}', 'nodes': '${env:DB_NODES,[]}'}})os.environ["DB_PORT"] = '3308'print(repr(cfg.database.port)) # converted to intos.environ["DB_NODES"] = '[host1, host2, host3]'print(repr(cfg.database.nodes)) # converted to listos.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'print(repr(cfg.database.password)) # kept as a string3308
['host1', 'host2', 'host3']
'a%#@~{}$*&^?/<'
xxxxxxxxxxYou can also set a default value for environment variables:xxxxxxxxxxconf = OmegaConf.create({"user" : {"age" : "${env:AGE, 42}"}})print(conf.user.age)42
Custom interpolations¶
xxxxxxxxxxYou can add additional interpolation types using custom resolvers. This example creates a resolver that adds 10 the the given value.xxxxxxxxxxYou can add additional interpolation types using custom resolvers.The example below creates a resolver that adds 10 to the given value.xxxxxxxxxxOmegaConf.register_resolver("plus_10", lambda x: int(x) + 10)conf = OmegaConf.create({'key': '${plus_10:990}'})print(conf.key)xxxxxxxxxxOmegaConf.new_register_resolver("plus_10", lambda x: x + 10)conf = OmegaConf.create({'key': '${plus_10:990}'})conf.keyxxxxxxxxxxYou can take advantage of nested interpolations to perform custom operations over variables:xxxxxxxxxxOmegaConf.new_register_resolver("plus", lambda x, y: x + y)conf = OmegaConf.create({"a": 1, "b": 2, "a_plus_b": "${plus:${a},${b}}"})conf.a_plus_b3
xxxxxxxxxxBy default a custom resolver’s output is cached, so that when it is called with the same inputs we always return the same value. This behavior may be disabled by setting `use_cache=False`:xxxxxxxxxximport randomrandom.seed(1234)OmegaConf.new_register_resolver("randint", lambda a, b: random.randint(a, b))c = OmegaConf.create({"x": "${randint:0, 1000}"})print("With cache:")print(f"c.x = {repr(c.x)}")print(f"c.x = {repr(c.x)}") # same as above thanks to the cacherandom.seed(1234)OmegaConf.new_register_resolver("randint_nocache", lambda a, b: random.randint(a, b), use_cache=False) # <== disable cache!c = OmegaConf.create({"x": "${randint_nocache:0, 1000}"})print("Without cache:")print(f"c.x = {repr(c.x)}")print(f"c.x = {repr(c.x)}") # not the same anymore since the cache is disabledWith cache: c.x = 989 c.x = 989 Without cache: c.x = 989 c.x = 796
Merging configurations¶
Merging configurations enables the creation of reusable configuration files for each logical component instead of a single config file for each variation of your task.
Machine learning experiment example:
conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)
Web server configuration example:
conf = OmegaConf.merge(server_cfg, plugin1_cfg, site1_cfg, site2_cfg)
The following example creates two configs from files, and one from the cli. It then combines them into a single object. Note how the port changes to 82, and how the users lists are combined.
xxxxxxxxxxbase_conf = OmegaConf.load('../source/example2.yaml')print(OmegaConf.to_yaml(base_conf))xxxxxxxxxxsecond_conf = OmegaConf.load('../source/example3.yaml')print(OmegaConf.to_yaml(second_conf))xxxxxxxxxxfrom omegaconf import OmegaConfimport sys# Merge configs:conf = OmegaConf.merge(base_conf, second_conf)# Simulate command line argumentssys.argv = ['program.py', 'server.port=82']# Merge with cli argumentsconf.merge_with_cli()print(OmegaConf.to_yaml(conf))xxxxxxxxxx